CHARACTERISTIC POLYNOMIALS 



OF SAMPLE COVARIANCE MATRICES: 
THE NON-SQUARE CASE 

H. KOSTERS 

Abstract. We consider the sample covariance matrices of large data matrices 
which have i.i.d. complex matrix entries and which are non-square in the sense 
that the difference between the number of rows and the number of columns tends 
to infinity. We show that the second-order correlation function of the charac- 
teristic polynomial of the sample covariance matrix is asymptotically given by 
the sine kernel in the bulk of the spectrum and by the Airy kernel at the edge 
of the spectrum. Similar results are given for real sample covariance matrices. 



1. Introduction 

The characteristic polynomials of random matrices have attracted considerable 
attention in the last few years, one reason being that their correlations seem to 
reflect the correlations of the eigenvalues IBHTI lBH2l lBH3l IF5^ IBDSI I5F1 IXFI 
imi\^ miKoTllKo2llKo3] . 

In this paper we continue with our investigation |Ko3j of the second-order corre- 
lation function of the characteristic polynomial of a sample covariance matrix. 
This function is given by 

/(n, m; fi, u) := E (det(Z(n, m) — fi) det(Z(n, m) — z/)) , (1-1) 

where n, m G N, /i, G M, and Z(n, m) is a complex or real sample covariance 
matrix defined as follows: 

Complex Sample Covariance Matrices. Let Q be a distribution on the 
real line with expectation 0, variance 1/2 and finite fourth moment 6, and for 
given n,m G N with n > m, let X := X{n,m) := (-'^ij)i=i,...,n;j=i,...,m denote the 
nxm matrix whose entries Xij are i.i.d. complex random variables whose real and 
imaginary parts are independent, each with distribution Q. Let X* = X*(n,m) 
denote the conjugate transpose of Z. Then the Hermitian m x m matrix Z := 
Z(n,m) := X(n,m)*X(n,m) is called the (unrescaled) sample covariance matrix 
associated with the distribution Q. 

Real Sample Covariance Matrices. Let Q be a distribution on the real 
line with expectation 0, variance 1 and finite fourth moment b, and for given 
n,m G N with n > m, let X := X{n,m) := {Xij)i=i^_^^^ri;j=i,...,m denote the nxm 
matrix whose entries Xij are i.i.d. real random variables with distribution Q. Let 
X'^ = X'^{n,m) denote the transpose of X. Then the symmetric m x m matrix 
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Z := Z{n,m) := X{n,m)^ X{n,m) is called the (unrescaled) sample covariance 
matrix associated with the distribution Q. 

We are interested in the asymptotic behavior of the values f{n]y,miy;iJ,]\f,U]\f) 
as — > cxD, for certain choices of the sequences (tin), (miy), (fj-N), (i^n)- 

In a recent paper |Ko3) we considered the "square" case where the difference 
ajv := nj^i — m]\f is fixed. We showed that in this situation, the second-order corre- 
lation function of the characteristic polynomial of a complex sample covariance 
matrix is asymptotically given (after the appropriate rescaling) 

- by the sine kernel in the bulk of the spectrum, 

- by the Airy kernel at the soft edge of the spectrum, 

- by the Bessel kernel at the hard edge of the spectrum. 

Moreover, similar results were obtained for real sample covariance matrices. 

The purpose of this note is to derive similar results for the "non-square" case 
where the difference := — rriN tends to infinity in such a way that the ratio 
7Ar := rriN/nN tends to some constant 700 G (0; 1) sufficiently quickly. We will show 
that in this situation, the second-order correlation function of the characteristic 
polynomial of a complex sample covariance matrix is asymptotically given (after 
the appropriate rescaling) 

- by the sine kernel in the bulk of the spectrum, 

- by the Airy kernel at the edge of the spectrum. 

Note that both edges of the spectrum are "soft" when 700 G (0; 1). 

These results are in consistence with results by Ben Arous and Peche |BP] . 
Tao and Vu |TVj . Soshnikov [So], Peche ^Pej, and Feldheim and Sodin 
|FS-lj . who obtained similar results for the (more relevant) correlation function 
of the eigenvalues, yet under stronger assumptions on the underlying distribu- 
tions. Ben Arous and Peche (BPj proved the occurrence of the sine kernel in 
the bulk of the spectrum for a certain class of complex sample covariance matrices 
with 7oo = 1. Very recently, TAG and Vu |TVj extended this result to a quite 
general class of sample covariance matrices, still with 700 = 1. Soshnikov |So) 
and Peche [Pel established the occurrence of the Airy kernel at the upper edge of 
the spectrum for real and complex sample covariance matrices whose underlying 
distributions are symmetric with exponential tails (or at least finite 36th moments). 
Recently, Feldheim and Sodin |FS-lj proposed another approach to obtain these 
results, which also works for the lower edge of the spectrum. Thus, our results add 
some support to the wide-spread expectation that correlation functions in random 
matrix theory are universal, and subject to very weak moment conditions only. 

It is well-known that the asymptotic distribution of the global spectrum of the 
rescaled sample covariance matrix (1/n) Z{n, m) is given (both in the complex case 
and in the real case) by the Marcenko-Pastur distribution of parameter 700 , which 
has the density 

5(0 :=;r^v/(e-c*)(e-e), e*<e<r, (1-2) 

where .f* := (1 — ^/Too)^ and ^* := (1 -|- -^7^)^. Note that as well as g 

implicitly depend on 700, but this dependence will be kept implicit throughout 
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this paper. Also, note that ^*,^* are the solutions to the equation 

^2-2(l + 7oo)e + (l-7oo)' = 0. (1.3) 

Unless otherwise mentioned, we will always assume that 1 < niN < un ■= N 
for all N E N, ajv := niy — tun oo as N ^ oo, and 7jv := mjv/riAr — Too 
as AT — oo, where 7oo G ]0; 1]. Our main results are as follows: 

Theorem 1.1. Suppose that = N and m^r = ^"700 + o{N) as N ^ 00. Let f 

denote the second-order correlation function of a complex sample covariance matrix 
satisfying our standing moment conditions. For any £ {$.*,(,*), ^, G M, setting 

Zn{^, fi, u) := {N^e + N^iii + iiuy"'^ exp (-iV^ - + z.)) , 

A := m/(7ooC/(0); ^ '■= ^l{loog{^)), we have 

-iry f{n,m-N^ + fi,N£, + u) 



lim (7003(0) ZN{^,fi,i') , , 

iv-»oo \ n\m\ 

= exp(fe*)S(/x,i.), (1.4) 

where b* := 2(6 — |) and S is defined by 

7r(x - y) 

Theorem 1.2. Suppose that rijv = N and = N^oo + o{N) as N 00. Let 
f denote the second-order correlation function of a real sample covariance matrix 
satisfying our standing moment conditions. For any ^ G {^*,^*), G M, setting 

Zn{^, fi, u) := {N^e + iVe(M + iy)+ /i^y)"^/' exp {-N^ -'^{i^ + u)) , 

A := l^/{lcog{0), ^ ■= ^/{loog{C)), we have 



hm iV ^ (7oo5(U) ZN{i,H,y) j — j 

i^oo \ n\m\ J 

= exp(r)i(fti.), (1.6) 



where b* := (6 — 3) and S is the sine kernel defined by 

2sin7r(x — y) 2cos7r(x — y) 
7r(a; - yY [x - yf 

Theorem 1.3. Suppose that = N and = Nj^o + o(N^/'^) as N ^ 00. 
Let f denote the second-order correlation function of a complex sample covariance 
matrix satisfying our standing moment conditions. For any /x, G M, setting 

Zn{^, fi, u) := (^fN^ + + v)iN^I-' + iiuN^'^'Y''''' exp (-^iV - i(/x + u)N^"') 

jl := y := e^'l^'^'^, we have 

lim ( C^/37- Ve ,VV3 z.iC, +A, +^) • fin,m;CN^^NyN + ONy^) \ 
N^oo \ n\m\ I 

= exp{b*)A{n,u) (1.8) 
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for (, = O'f^d, when 700 / 1, 

AT^oo \ n\m\ I 

= exp(6*)A(/i,z^) (1.9) 

/or 1^ = where b* := 2(6 — |) anc? A is t/ie ^iry kernel defined by 

A(,.,),,AiWAi'fa)-Ai'MAifa)^ (1,10) 

X - y 

Theorem 1.4. Suppose that = N and rriN = Nj^o + o{N^/^) as N ^ 00. Let 
f denote the second-order correlation function of a real sample covariance matrix 
satisfying our standing moment conditions. For any ^, G M, setting 

ZN{i,pi,v) := fe^7V^ + (/i + i/)eiV^/3^/xi^A^2/3\"^/2 expf-^iV- i(^ + i/ 



:= C^^^Too^^*^/^, i> := C'^^^^oo^^^i^, we have 



r (, -1/27 (c f{n,m;^N + fiNy\^N + uNy^) \ 

lim U 7oo ^ Zn[£., +1^) j — j 

\ n\m\ I 

= ex.p{b*)A{fi,u) (1.11) 

for C = C o,nd, when 700 / 1, 

lim UToo ^ ZN[i, -H, -I') j — j 

Af-*oo \ n\m\ I 

= ex.p{b*)A{j^,v) (1.12) 

for = where b* := (6 — 3) and A is defined by 
2 Ai(x) Ai'{y) - 2 Ai'(x) Ai(y) 



A(x,y) := 



(x - y) 
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, (x + j/)Ai(x)Ai(j/)-2Ai^(x)Ai^(j/) 

+ /NO • 

(x - yy 



Note that in all the cases, the limit is almost universal in that it depends on 
the underlying distribution only via a multiplicative factor containing the fourth 
cumulant. Also, note that we obtain the sine kernel in the complex setting not only 
for 7oo = 1 but also for 700 < 1. 

This paper is organized as follows. Section 2 contains the proofs of Theorems 
|l.l| and |l.3| (Theorems 1.2 and 1.4 may be proved similarly.) In Section 3 we state 
some auxiliary results concerning Bessel functions which are needed for the proofs. 
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2. The Proofs of Theorems 11.11 and 11.31 



The starting point for the proofs of Theorems 1 1 . 1 1 and 1.3 wiU be the fohowing 



exponential-type generating function for the second-order correlation function of 
the characteristic polynomial of the (unrescaled) sample covariance matrix satis- 
fying the moment conditions from Section 1: 

Proposition 2.1. For any a G N, l^j < 1, 

f{m + a,m-^i,v) ^ _ (-(^+'^)t^ + b* z) • (g^^^) 

{m + a)\m\ ^ (1 - z)i+(2//3) . ((;xi/z)V2)" ' ^'^ 



E 

r?i=0 



where b* := 2(6 — |) and [3 := 2 in the complex setting, b* := 6 — 3 and /? := 1 in 
the real setting, and la denotes the modified Bessel function of order a. 

Note that {w /2)~°'Ia{w) is an even entire function of as follows directly from 
the power series representation of the modified Bessel function. Thus, the right- 



hand side in (2.1) extends to a meromorphic function on the complex plane, with 
a singularity at z = 1. In the subsequent calculations, we will always consider the 
principal branch of Ia{w). 

We only give the proofs for the complex setting, the proofs for the real setting 
being very similar. 

We will always assume that on '■= — rriN oo and 77V := miy/nj\f — > 7oo 
for some constant 700 G ]0, 1]. (The case where a^v remains bounded is already con- 
sidered in |Ko3) .) Furthermore, we will usually omit the index of the parameters. 



and also of several auxiliary functions defined below. By Proposition 2.1, we have 
the integral representation 

f{n,m;fi,u) 1 f exp (-(/i + i^) + 6*z) • (^^f^^) 



n!m! (1 - z)2 • ((/xi.z)i/2)a ' ^^'^^ 

where a denotes a contour around the origin. Similarly as in |Ko3] . we will show 



that the main contribution to the contour integral in (2.2) comes from a small 
neighborhood of the point zq = 1. To determine the asymptotic behavior of 
the values f{nN,TnN,IJ'N,i^N) in Theorems |1.1| and 1.3, we proceed similarly as 



in |Ko3] ■ but we have to address some additional complications arising from the 
circumstance that oat ^ 00. To deal with the this problem, we will use various 
well-known uniform asymptotics for the modified Bessel function of large positive 
order (see e.g. Chapters 10 and 11 in Olver [Or). For convenience, we state these 
results (and several bounds deduced therefrom) in Section 3 of this paper. 

We will choose the contour a = ajy consisting of the following parts = aN,i, 
i = -2,...,+2 (see Figure [TJ: 

• a-2{t) ■■= exp(it), -TT < t < -ia/N'', 

• a-i{t) := (1 - t/N'^) exp{-ia/N'^), 0<t<l, 

• ao{t) := (1 - 1/N'') exp(it), -ia/N'' < t < ia/N'', 

• a+i{t) := (1 + t/N'^) exp(ia/iV^), -1 < t < 0, 

• a+2{t) := exp(it), ia/N"^ < t < vr. 

Here a > is a positive real number which will finally be chosen sufficiently large, 
and rj := 1 for the bulk of the spectrum and 77 := 1/3 for the edge of the spectrum. 



Characteristic Polynomials of Sample Covariance Matrices 



6 




Figure 1. The contour a. 



The rough idea behind the choice of this contour is the following: After the 



appropriate rescaling, the main contribution to the integral (2.2) comes from the 



part do, for which we can use the uniform asymptotics (3.1 ) for the modified Bessel 



function in the positive half-plane. The other parts are essentially error terms, 
which can be made arbitrary small by picking a and N sufficiently large. However, 
to prove this, we need rather precise approximations and bounds for the modified 
Bessel function close to the imaginary axis, and this is our main motivation for 
the choice of the above contour. For the contours cr_i and cr+i, we can use the 



uniform bound (3.3) for the modified Bessel function near the imaginary axis, but 



away from the turning points ibi. For the contours (T_2 and (T_|_2, we can use the 



uniform bounds (3.6) and (3.7) for the modified Bessel function on the imaginary 



axis, including the vicinity of the turning points ibz. 

In the following proofs, we adopt the convention that all asymptotic bounds 
may depend on the sequence {oiis[)Nen as well as on the "shift parameters" ^, /i, v, 
unless otherwise indicated. A similar remark applies to large positive constants, 
typically denoted by C. 

After these comments, we turn to the proofs of the main theorems. 

Proof of Theorem \l.l\ Let ^, ^u, v be as in Theorem and let a > 1. We assume 
throughout this proof that is sufficiently large. By ( |2.2[ ) and the definition of 
Zjv(^, /i, z^), we have 



where 



h{z) = hN{z) :-- 



f{n,m;Ni + ^i,Ni + v) 1 



n\ ml 



27ri 



h{z] 



dz 



(2.3) 



exp (-(A^e + + '^))T^f + b*z) ■ la (wiz)) 

(1 _ z)2 . ^(n+»n)/2 



w{z) = wn{z) := v) + ■ 
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and a = is the contour specified above, with 77 := 1. Note that a imphcitly 
depends on a. 

For any a G [1, 00], let 



5(a) := 



exp( 



(6*) r+« exp (/ioo(e)(l - in) - - - 



47^3/2^1/2 J^^ 



(1 - iu)3/2 



du , 



where /ioo(0 — "i? + 5(1 + 7oo) - i(l - 7oo)^r^- Note that by ([Ts]) we have 
hooiO > for ^ G ^*]. We will show that for any 6 > 0, there exists some a > 1 
such that the following holds: 



lim 

iV— >oo 27ri 



lim sup 



lim sup 



/ Kz) 


dz 




z 


[ Kz) 


dz 






z 


[ Kz) 


dz 




J (72 


z 



S{a), 



< 6. 



(2.4) 
(2.5) 
(2.6) 



Then, by symmetry, similar bounds hold for the integrals along a-i and cr_2. Using 
these results, it is easy to see that 

1 



lim 



dz 

h{z) — = ^(cxd) . 



Now, by Laplace inversion, we have, for t > 0, a S M, 

1 f^e-'^'/^^ , 2sm(aVt) 

e dz = 

i-ioo z^^^ 



2iTi 



(see e.g. p. 245 in [Er] ) . We therefore get 



5(00) = i exp(6*) (/ioo(e)/e)'/' 



sinf(^-z.) 



{f^-^){hoo{0/0'^' 



Now replace the local shift parameters /i,z/ with /^/(7oo5'(C))) ^/(7oo9(0) 

1 /2 A k 

observe that (/ioo(?)/0 = ^7oo<?(0 (which is a simple consequence of (1.3|) 
to obtain (IlI). 



To prove (2.4 1 - (2.6), we write 

h{z) = 

where 



hi{z)h2{z) 



and 



(1 - Z)2 2(«+m)/2 ' 

hi{z) :=exp^-(A^^ + i(/i + z^))^^ + 6*z^ •exp(+i«(z)) 
h2{z) := exp {-w{z)) ■ la {w{z)) . 
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Since 



i{z) = 2 (iVe + + ^)- K/x - vf/Ni + 0{l/N^)) ^ 
a simple calculation yields 



hi{z) = exp - iV^ 



1 



z 1 , . 1 



1 + 



' \{^,-^f/{Ni)^+b*z 



+ 



1 - z 



1 - z 



(Recall our convention that implicit constants in O-terms may depend on {um) 
V, which are regarded as fixed). 



We first prove (2.4). In doing so, we use the notation Oa to denote a bound 
involving an implicit constant depending also on a (in addition to (otv); ^jMjZ^)- 
Let R = Rn '■= (1 — 1/^)- Substituting z = -Re** and t = u/N in the integral on 
the left-hand side in (2.4), we obtain 



du . 



(2.7) 



1 /ii(i?e^"/^)/i2(i?e*"/^) 

2^ (1 - i?e™/JV)2 . (^gm/^)(n+m)/2 

Now, using Taylor expansion, we have the following approximations, for \u\ < a: 
/ii(i?e^"/^) = exp ( - iC(l - iu) - ^(m - ufc'/i^ - ^n) + b*) (1 + o,(l)) 

(1 _ i?e™/^)2 = (1 - m)yN^ (1 + 0,(1)) , 



For the second approximation, we have also used the uniform asymptotics (3.1 ) for 
the modified Bessel function. Indeed, since 

aiV = (l-7oo + o(l))iV, 
y;(i?e^"/^) = 2iV2^/(l - iu) (1 + o,(l)) , 



and 



y/l + z^ + log ■ 



^z~^ + 0{z- 



{21 



1 + Vl + ^2 

for \z\ oo, I argzl < ^tt, it follows from ( |3.1| ) that 

/i2(i?e*^/^) = exp(-'u;(i?e^^/^)) • /„('u;(i?e*^/^)) 
exp(-iaVt.(i;eWA^)) , 

= ^ (1 + Oa(l)) , 

V2^«;(i?eW^) ^ ' 

whence the approximation for /i2(iie™/^). Inserting the above approximations 
into (2.7) and recalling the definition of S{a) yields (2.4). 
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We now prove (2.5) and (2.6). For z £ range (fii) U range (CJ2), write ^/z = re**^ 
with r £ [{1 — 1/Ny^; 1], tp £ [a/2N; '/r/2]. Since a > 1, we then have the estimates 



Re 



1 + 2r cos if + r"^ 
1 

< - 



Re 



Ini 



(r — r'^) cos if 
1 — 2r2 cos 2(p + 

(r + r^) sin ip 



= 0{N) 
>0, 



> 0. 



(2.9) 
(2.10) 
(2.11) 
(2.12) 



l-2r2 cos2(^ + r4 

In particular, (2.9) and ( 2.10| ) show that hi{z) is uniformly bounded. Moreover, 

|^-(n+m)/2| ^ ^-j^ _|_ 2/A^)("+™)/2 = (1 + 2/iV) I (^+'^°°+°(^))^ = 0(1) . 



The proof of (|2.5|) and (|2.6|) is therefore reduced to showing that for any 6 > 0, 

h2{z) 



there exists some a > 1 such that 
h2{z) dz 



lim sup 



(l-z)2 



lim sup 



0-2 



(1-z)^ 



< 5. 



(2.13) 



For the first integral in (2.13), note that for t G [0; 1] and fixed a > 1, we have 



aiV = (l-7oo + o(l))iV, 



and 



w ((1 - t/N) e^"/^) = 2N^a{t - ia) (1 + o,(l)) . 



Writing z = {1 - t/N)e''^/^ with t E [0; 1] and using ( |2.11[ ) as weh as 

the uniform bound ( |3.5[ ) for the modified Bessel function, it therefore follows that 
for all sufficiently large N £N (depending on a). 



\h2{z)\ = |exp(-w(z)) • Iaiw{z))\ < 



C lexp 



< 



/ 1 ; — TT V-L -|- Ofi 

y2^K^ 



(1)) 



where C is a positive constant which does not depend on a and which may change 
from step to step in the sequel. Combining this with the estimates |1 — > 
a/N{l + Oa(l)), 1^1 > 1 - 1/A^, it follows that 

dz ^ C7(l + a2)i/4 

z ~ 



lim sup 



(1-Z)2 



/47r| 



Since for a > 1 sufficiently large, the right-hand side is clearly bounded above by 6, 
this proves (2.5 ). 



For the second integral in (2.13), note that for \z\ = 1, we have Re w{z) =0 and 
Im w{z) > from (2.11) and (2.12). Thus, we may use the bounds (3.8) and (3.9) 
for the modified Bessel function on the imaginary axis. Writing z = e** with 
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t G [1/-^; 71"], we have, for all sufficiently large N gN, 



10 



and 



\wn{z)\ = 2y^N^^ + N^{fi + u)+fiiy 



\wN{z)/aN\ > 



(1-7^)|1 



> 



1 - z 



II - ( 



> 



11 



> 



Hence, by ( |3.8[ ) and ( |3.9[ ), for all sufficiently large N eN, 
\h2{e'')\ < 



C|i-e^*|i/2/Ari/2 ; |t| < (logiV)-i 

(^|l_git|l/3/^l/3 ; |t| > (logiV)-l 



where C is a positive constant which depends only on ^ and which may change 
from occurrence to occurrence in the sequel. It follows that 



h2{z) dz 



/i2(e**) 



a/N (1 - e 



it\2 



dt 



< 



< 



< 



C 

c 
c 



(logiV)-i ^ 

(logiV)-i ^ (J 
dt + 



dt + 



a/N 

[N/a 



,1/2 



t3/2 

c 

+ 



Ari/3 



C 

iVV3 



(logAf) 



1 



(log7V)-i |l-e^*|5/3 
1 



-1 t5/3 



(logA^)^/^ 



Ari/2 ^ ' ^ ATi/s 

Since for a > 1 sufficiently large, this is clearly < 5 for all sufficiently large G N, 
this proves (2.6 ). 



The proof of Theorem 1.1 is complete now. 



□ 



Proof of Theorem 1.3. Let ^ G l'^*,'^*}, l^-,^ G and a > 1. When we use 
the symbols it and ^, we mean the upper sign for i = ^* and the lower sign 
for ^ = We assume throughout this proof that N is sufficiently large. Again, 
by (2.2 ) and the definition of Zn{(,, fJ-, z^), we have 



nl ml 



2m 



h{z] 



dz 
(2.14) 



where 



h[z) = hN{z) :-- 



exp (-(^iV ± liii + i/)iVV3) i±| + b*z^ ■ 

(1 _ 2)2 . ^(n+m)/2 



(z) = Wn{z) := 2a/^2^2 ± + zy)^iV4/3 + ^z,iV2/3 



1 - Z 



and (T = (Ttv is the contour specified above, with rj := 1/3. Note that a implicitly 
depends on a. 
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For any a £ [1, oo], let 



4^3/2^1/2 



We will show that for any 6 > 0, there exists some a > 1 such that the following 
holds: 



hm / h{z) — = A{a) , 

n^oo 27ri J^^ z 



lim sup 



lim sup 



iVV3 / h{z) 



iVV3 / h{z) 



dz 



dz 



<5. 



<6. 



(2.15) 



(2.16) 



(2.17) 



Then, by symmetry, similar bounds hold for the integrals along a-i and a -2- Using 
these results, it is easy to see that 

lim — [ hiz) — = A(oo) . 

Substituting z = {ihooY^^ dz = [^/^ocY^^dz, and shifting the path of integration 
back to the line 1 + (which is easily justified by Cauchy's theorem), we obtain 



^(oo) 



7oo 



1/6, 



exp(6*) /-i+^^^exp 



4^3/2^2/3^ 7i 



(_^,3 _ i(^ + y)iJ.'i~V^z - - uf^U'^~^/'z-^) 



Making the replacements ji ^ (C^^'^/7(^^)/U, u 
by ('?^''^/7oo^), we further obtain 



^3/2 



dz . 



7oo )i' and multiplying 
By the integral representation for the Airy kernel (see e.g. Proposition 2.2 in |Ko2j ) . 



the latter expression is equal to exp(6*) A(/i, z^), whence (1.8) and (1.9) 



To prove (2.15) - (2.17), we proceed similarly as in the preceding proof. To 

hi{z) h2{z) 



begin with, we write 



h{z) 



where 

hi 

and 



(1 - z)2 z("+™)/2 ' 

{z) :=exp(^-(CiV±i(^ + z.)Ari/3)l±| + 6*z^ •exp(+u;(z)) 
h2{z) := exp {-w{z)) ■ la {w{z)) . 
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Similarly as above, we then have 



+ 6*z+^0(l/iV)^ . 

Starting from this representation, the proof is by and large similar to the preceding 
proof. However, a notable difference is given by the fact the leading-order terms 
cancel out now, which is why we have to keep track of several additional terms in 
the asymptotic approximations. 

We begin with the contour integral for ctq. Let R = R^ ■= (1 — 
Substituting z = i?e** and t = u/N^/'^ in the integral on the left-hand side in (2.15 ), 
we obtain 

1 /■+" hi{Re'^l^"')h2[Re'^'''"") 

Similarly as in the preceding proof, we use the notation Oa to denote a bound 
involving an implicit constant depending also on a (in addition to (otv); CjA*)^)- 
Then, putting := 1 — 7oo for abbreviation and using Taylor expansion, we have 
the approximations, for \u\ < a, 

h,{^R^^u/N^/') = exp f - 1(1 - iu)iN^/^ - kiV^s _ 1 ^ + 1(1 _ ^^)3^ 



T |(^ + -iu)-\{^i- vfC\l - iu)-^ + b*) (1 + Oa(l)) , 



96 '^00 



(1 _ Re^-/^'^'f = (1 - ^uf/N'/^l + Oa{l)) , 

^j^^ru/N^ym+n)/2 ^ ^_ l^^ ^ ((^ _ .^)^2/3 + 1^1/3 + 1)) (l + o,{l)) . 

For the second approximation, we have used the uniform asymptotics (3.1) for 
the modified Bessel function. Indeed, since 

aN = {I - loo) N + oiN^^) , 
wiRe'^"""") = 2N^l^i/{l - iu) (1 + 0,(1)) , 

and 



\/rT^ + log == = z - \z-^ + + 0(z-5) 

1 + V 1 + 

for \z\ oo, I argz| < ^vr, it follows from ( |3.1| ) that 
/i2(i?e*"/'^'^') = exp(-^i;(iie''"/^'^')) • /,(ii;(iie™/^'^')) 

exp (-iaVu;(i?e-/^'^') + ^^a^ lw\Re'^ I 



^2t^w{R(^^I^^^^) 



(l+Oa(l)) . 
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Now, by straightforward expansion, 

Plugging this into the preceding expression and using once again the assumption 
a = (1 — 7oo)-^ + o{N^^^) yields the approximation for /i2(i?e*"/^^''^). 

Inserting the above approximations into (2.18) and ordering with respect to 
fractional powers of N, we obtain 

exp(6*) exp(r) 



47^3/2^1/2 (l-m)3/2 



(l + Oa(l)) dt 



with 



T = + 
+ 
+ 



1(1 _ iu)^ + 1(1 + ^^)(i _ iu) - - in)r')iV'/' 

1^2 

8 lo 



+ ^(1 - iufC T + -iu)-l{fi- ufCHl - iu)-^ 



By the characterizing equation (1.3) for the edge of the spectrum, we have 

-ie-i(i-7oo)Ve + Ki + 7oo) = o. 

Thus, the terms in the large round brackets cancel out, and the remaining sum 
can be simplified to 

T = (3^700(1 - iuf ± + i/)(l + 7oo - 6(1 - iu) - - - iu)-^) /e . 

Recall that = (1 ± ^700)^ at the upper edge and the lower edge of the spectrum, 
respectively. Plugging this into the expression for T and simplifying, we obtain 

T = (i^7oo(l - iuf - 1 V7^(m + - iu) - - - , 

and the proof of (2.15) is complete. 

The proof of ( |2.16| ) is similar to that of ( |2.15[ ). Setting z := (l-t/TV^/^) exp^ia/N^/'-^) 
with t G [0; 1] and assuming that G N is large enough, we have the following bounds: 



l^i(^)l < 



\h2{z)\ < 



xp [-\{t- ia)eiV2/3 _ ii2^^i/3 _ _^^3^ +^{t- iaf^ 
T lifi + u){t - ia) - lifi - vfC\t - la)-^ + 6*) (1 + Oa{l)) 



C 



exp ( - \lUt - ^a)C'N^^' - llLt'C'N'^' - ^.^It'C' 



— ia 

- m^lit - iafr' ± -sllif^ + ^){t- m)r' + TM^Lit - iafr"") ■ (l + Oa(l)) 
\l-z\^>\^t-ia)yN^/\l + Oa{l))\, 

> I exp (-1(1 + 7oo)((t - ia)N'/' + It^N'/'^ + It')) (l + o,(l)) | . 
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Let us comment on the second bomid (the others bounds being straightforward) . 
The uniform asymptotic approximation ( |3.1| ) is not apphcable anymore, since w{z) 
approaches the imaginary axis as f — > 0: 

w{z) = 



However, we may use the uniform asymptotic bound (3.5) instead, since Reit;(z) 
> 0, as follows from a similar estimate as in (2.11 ). We thus obtain, for all G N 
sufficiently large (depending on a). 



h2iz) 



< 



exp{-w{z)) ■ Ia{w{z)) 

c 



exp 



■\/2ttw{z) 

where C > is a constant not depending on a. Since 
1 



/w{z) + j-^aVw\z)){l + Oa{l)) 



w{z) 



this entails the asserted bound by substituting \/w{z) and simplifying. 



Putting it all together and doing similar simplifications as in the proof of (|2.15|), 
;ral on th( 

Cexp(6*) 



we find that the integral on the left-hand side in (2.16) is bounded above by 

"1 |exp(r)| 



3< 7o |t-ia|3/2 



(1 + 0,(1)) dt, 



where C is a positive constant which does not depend on a (and which may change 
from step to step as usual) and 

T = (3^7oo(t - iaf + iy)^{t - ia) - ^(^ - i^f{t - ia)-^) . 

The real part of this expression is obviously of order 0(1). It therefore follows that 

C 



lim sup , 



exp(T)| , , , Ca-3/2 

^' (l+Oa(l)) dt< 



Since for a > 1 sufficiently large, the right-hand side is clearly bounded above by 6, 
this proves (2.16). 



It remains to prove (2.17). Using similar estimates as in (2.9) and (2.10), we see 

|/i2(e**)| 



that it suffices to show that 



(/•TV 11 
Ja/m/'A |1 



oit\2 



dt \ <5 



(2.19) 



for a > 1 sufficiently large. Moreover, by similar estimates as in (2.11) and (2.12), 
we have Re w{e^^) > and Im w{e'^^) > for all t £ [a/A^^/^; vr], so we can use 



( |3^ and to bound /„(u;(e**)). 

Let £ G (0;l/3) be a small constant which will be chosen later, let / denote 
the subset of those t £ [a/N^/^;Tr] such that Im w{e'^^)/aN G (1 — e; 1 + e), and 
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let J denote the complement of this subset. Using the stronger bound (3.9) on J 
and the weaker bound (3.8) on /, we then have, for sufficiently large G N, 

1 



a/iVi/3 



11 - e**p 



dt < CN-^I'^ 



11 



t|3/2 



dt + CN-^I^ 



1 



11 



t|5/3 



dt 



< C7^iV-i/2(^i/3/^)i/2 ^ C7iV-i/3j^(j)/(inf 7)5/3 ^ 

Here denotes the length of the interval I, C is a constant which depends 

only on ^, and is a constant which may additionally depend on e. (Of course, 
both constants may change from occurrence to occurrence as usual.) Clearly, once 
e > is fixed, we may make the first term arbitrarily small by choosing a > 
sufficiently large. Hence, to complete the proof of (2.19), it remains to establish 
an appropriate bound on the second term, i.e. on 

1(1) /(inf 1)5/3 _ ^j^ig 

purpose, 

first note that 

Im u;(e'*) = iVC(sin(t/2))-^ (1 + o(l)) 

and therefore 



Im w{e )/aN 



-(sin(t/2))-i(l + o(l)). 



1 - 7Ar 

For the upper edge of the spectrum, we have ^ > 1, which implies that the interval 
I is empty for e sufficiently small and sufficiently large, and (2.19) is proven. 
For the lower edge of the spectrum, we have < ^ < 1 as well as ^ / (1 — 700) < 1 
(as is readily verified) and therefore, for e sufficiently small and N sufficiently large. 



/ c 



a/N 



1/3. 



TT 



n 2 arcsin 



,(l-7oo)(l + 2e)' (l-7oo)(l-2e). 

In particular, inf / > c for some positive constant c depending only on ^ and 700- 
Moreover, X{I) can be made arbitrarily small by picking e sufficiently close to zero. 
Thus, (2.19) is also proven. 



The proof of Theorem 1.3 is complete now. 



□ 
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3. Some bounds for Bessel functions 

In this section we state some uniform asymptotic approximations for Bessel 
functions from the hterature (see Chapters 10 and 11 in Olver |01| ) . Moreover, 
we deduce several asymptotic bounds for Bessel functions which are sufficient for 
controlling the error bounds in the preceding proofs. These asymptotic bounds 
should be well-known, but we have not been able to find explicit statements suiting 
our purposes in the literature. Throughout this section, we assume that a G N 
and a. — > oo. 

We need the following uniform asymptotic approximations for Bessel functions, 
which can be extracted from Sections 10.7 and 11.10 in Olver |01| , respectively: 

Proposition 3.1. For any e > 0, 

/„(az) = -^-^^P^(l + a(l/a)) (a-oo), (3.1) 

where 

rj:= v^l + ^2+iog (3.2) 

1 + vl + ^^'^ 

and the O^-hound holds uniformly in the set € C : | arg2:| < |vr — e} . 
Proposition 3.2. For any e > 0, 

+ ^'^^^S^O.({l + |C|'''^)-')} (o-oo), (3.3) 

where 

1 + Vl 



z 

and the Oe-bounds hold uniformly in the set {z € C : \ arg z\ < it — e} . 



We refer to Sections 10.7 and 11.10 in Olver [01] for a discussion of the choices 
of the various branches. 

By Proposition 3.1 we have the following bound on /^(az), uniformly in the set 
{z € C : \z\ > 1 + e A | arg < jvr - e}: 

Using Proposition 3.2 it can be shown that this bound in fact remains valid up to 
the imaginary axis: 

Lemma 3.3. For any e > 0, 

laiaz) = Oe (^^) (a ^ oo) , (3.5) 



where rj is defined as in ( |3.2[ ) and the Os-bound holds uniformly in the set 
{z £ C : \z\ > 1 + e A | arg < ^Tr}. 
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Proof. Fix e > 0, and consider Ja{az) with a G N, z G := {z G C : |2;| > 1 + e 
A |argz| < vr — e}. We then have the uniform asymptotic approximation (3.3). 



We now use the following asymptotic approximations for the Airy function and its 
derivative, whicl 
in Olver [OI]): 



derivative, which hold for \z\ oo, | argz| < %t\ — e' (see Chapters 11.1 and 11.8 



Ai {-z) = ^-1/2 z-i/^l cos(|z3/2 _ ^ ^ 0,,(z-3/2)) 



Ai'(-z) = ^-1/2 z+i/^l sin(|z3/2 _ ^ ^ 



■3/2n 



cos(oZ' 



2 ^3/2 _ 1 



7r)z-3/2 (l + 0,,( 



.-3/2^ 



Using the inequalities |cos2;| < exp(|Im |sinz| < exp(|Im z|), and the fact 
that Ai, Ai' are holomorphic functions, it follows that 



Ai 



Ai'( 



Oe'\Z 



^1/^ exp(|Im |z3/2| 
'1/4 exp(|Im 1^3/2 1 



uniformly in z G C such that \z\ > e' and | arg z] < Ivr — e'. 

It can be checked that for a G N, z G Mg, | — a^/Sj^j > a,nd arg(— a^/Sj^) < 2^ _ 
for some e' > depending only on e > 0. Hence, 

Ai {a^'\) = Oe (a-i/'^C-O-'/'expdlm iai\)) , 

Ai'(a2/3^) = Oe (a+i/^(-C)+i/^exp(| Im ia^|)) , 
uniformly in z G M^, where C, = C,{z) is defined as in Q and e = i{z) := fC^/^. 



Inserting this into (3.3), it follows that 

^ / X /expfl Re (z)|) \ 

Uaz) = Oe ( (^,)i/2 J (a - oo) , 

the Cg-bound holding uniformly in z G M^. 

Thus, for z G {z G C : |z| > 1 + e A | argz| < jTt}, we have 

'exp(| Re a^(— iz)|) 
(az)V2 

when Im z > and 

'exp(|Re a^(+iz)|) 



\Ia{az)\ = \ Ja{-iaz)\ = Oe 



(a oo) , 



|/a(az)| = \Ja{+iaz)\ = Oe 



(az)^/^ 



{a 



oo 



when Im z < 0. A comparison of the definitions (3.2) and (3.4) (together with 
a careful discussion of the choice of branches) shows that ^(=piz) = — r/(z) it ^-iri , 
which completes the proof of the lemma. □ 
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The next lemma gives some upper bounds for the (unmodified) Bessel function 
on the positive real half- axis: 

Lemma 3.4. There exists a constant C > such that for all sufficiently large 

a en, 

c 



\Ja{ax)\ < 



for all X G (0, oo) . 



(3.6) 



More precisely, for any e > 0, there exists a constant > such that for all 
sufficiently large a G N, 

Ce 



\Ja{ax)\ < 



\ax 



1/2 



for all X G (0, 1 - e) U (1 + e, oo) . 



(3.7) 



These bounds are not the best possible, but they are sufficient for our purposes 
In terms of the modified Bessel function, they read 

C 



\Ia{iO(x)\ < 



\ax 



1/3 



for all X G (0, oo) 



and 



\Ia{iax)\ < 



for all X G (0, 1 - e) U (1 + e, oo) 



respectively. This is the form in which they have been used in the last section. 



(3.8) 
(3.9) 



Proof of Lemma 3.4 We start from the uniform asymptotic approximation (3.3), 
which we now consider for x > only. Note that C = C(^) is a decreasing function 
of X with C > for x < 1 and C < for x > 1. There exists a constant C > 
such that 



\Ja{ax)\ < C 





[ |Ai(a2/3c)| 


1 — x^ 


\ ai/3 



+ 



a5/3(i + |(|i/2) j 

for all X > for all sufficiently large a G N. We now use the bounds 



(3.10) 



Ai (+x) = Oix 



-1/4 



exp( 



Ai'(+x) = 0(x+^/^ exp(- 



.2^3/2 



X-)) 



)) 



(x +00) , 
(x +00) , 

Ai (-x) = 0(x~^/^) (x^+00), 

Ai'(-x) = 0(x+^/^) (x^+00), 

which follow from well-known asymptotic approximations for the Airy function and 
its derivative (see e.g. Section 11.1 in Olver |01j 1 . In particular, these bounds 
imply that Ai(x) = 0{{\ |x|i/^)~i) and Ai'(x) = 0(1 |x|^/^) throughout 
the real line. It therefore follows from ( |3.10 ) that there exist constants C", C" > 
such that 

\Ja{ax)\ < C" 



4C 




1 -x2 


1/4 1 



a 



(l + aV6|^|i/4) 

/3(l + aV6|(^|l/4) ' a4/3(l+al/3|(^|l/2) 



+ 



< C'" min 



1 



1/4 



a 



1/3' 



1/4 



a 



1/2 



for all X > for all sufficiently large a G N. 
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To deduce (3.6), use the first term inside the minimum and observe that since 
ICI ~ for X — > oo, Id ~ || logxp/^ for x — > 0, and C is an analytic function 

of X G (0,cx)) with C(l) = (see e.g. Section 11.10 in Olver [QI]), 

1/4 

= Oix''^-^) (0 < X < oo) . 



4C ^ ...^,-1/3, 



1 — x^ 

To deduce ( |3.7| ), use the second term inside the minimum and observe that 
for X (1 - e, 1 + e), |1 - x2|-V4 = 0^(x-i/2). □ 
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